Skip to content

feat(retention): chain-aware retention and pruning#8

Merged
nixrajput merged 2 commits into
mainfrom
feat/phase-g-retention
Jun 25, 2026
Merged

feat(retention): chain-aware retention and pruning#8
nixrajput merged 2 commits into
mainfrom
feat/phase-g-retention

Conversation

@nixrajput

@nixrajput nixrajput commented Jun 25, 2026

Copy link
Copy Markdown
Owner

Summary by CodeRabbit

  • New Features
    • Added chain-aware, retention-policy-based pruning (base + incrementals treated as a unit) with support for keep-last, max-age, and GFS-style tiered retention.
    • Added retention: configuration with defaults plus per-profile overrides, including precedence and “keep everything” behavior for empty/zero policies.
  • Bug Fixes
    • Prevents orphaned incremental dumps by pruning chains atomically and stopping at the first deletion failure within a chain.
  • Documentation
    • Updated README and added retention documentation, including dry-run-by-default and --apply deletion behavior.
  • Tests
    • Added coverage for chain-failure stop behavior and pruning outcomes in dry-run/apply/profile modes.

- Add a pure retention engine in internal/dumps (GroupChains + Plan, no
  I/O or clock): the catalog is grouped into restorable chains (a base
  plus its incrementals) and kept or pruned as a unit, so an incremental
  is never orphaned from its base. A chain's age is its newest member,
  so an actively-appended chain is never pruned mid-life.
- Support three composable rules with union semantics (a chain is kept
  if it satisfies any active rule, so a rule can only ever protect):
  keep_last N, max_age duration, and GFS daily/weekly/monthly. An
  all-zero policy keeps everything — prune is a no-op until configured.
- Add app.Prune: scopes to a profile, plans over chains, and on --apply
  deletes leaf-inward (incrementals before base) so an interrupted prune
  leaves at worst a complete shorter chain. Per-dump failures are
  collected, not fatal; the run reports them and exits non-zero.
- Rewire siphon prune to the engine: --profile, --keep-last, --max-age,
  --gfs-daily/weekly/monthly, dry-run by default, --apply to delete.
- Add a retention config block (defaults + per-profile override that
  replaces wholesale) with fail-fast validation; precedence is CLI flags
  > profile > defaults > keep-everything.
- Document in docs/RETENTION.md; update README and CHANGELOG. Unit-test
  the engine (GFS bucketing, union, chain grouping, keep-everything),
  the executor (leaf-inward order, scoping, collected failures), and the
  config (validation, override precedence).
@nixrajput nixrajput self-assigned this Jun 25, 2026
@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

An error occurred during the review process. Please try again later.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0a7e130c-8411-439c-971a-ca60b448a9a9

📥 Commits

Reviewing files that changed from the base of the PR and between ce22097 and 1a4a69d.

📒 Files selected for processing (8)
  • README.md
  • docs/RETENTION.md
  • internal/app/prune.go
  • internal/app/prune_test.go
  • internal/cli/dumps.go
  • internal/config/config.go
  • internal/dumps/retention.go
  • internal/dumps/retention_test.go
✅ Files skipped from review due to trivial changes (2)
  • README.md
  • docs/RETENTION.md
🚧 Files skipped from review as they are similar to previous changes (3)
  • internal/app/prune.go
  • internal/cli/dumps.go
  • internal/config/config.go

📝 Walkthrough

Walkthrough

This PR adds chain-aware retention planning, config-driven prune policy resolution, and CLI/app execution for dry-run or apply pruning. It also updates the README, changelog, and retention docs to describe the new policy shape and prune behavior.

Changes

Retention pruning flow

Layer / File(s) Summary
Chain model and grouping
docs/RETENTION.md, internal/dumps/retention.go, internal/dumps/retention_test.go
GFSPolicy, RetentionPolicy, Chain, RetentionPlan, and GroupChains define chain units and ordering, with tests for chain construction and orphaned incrementals.
Planning rules and GFS
internal/dumps/retention.go, internal/dumps/retention_test.go, docs/RETENTION.md
Plan combines keep_last, max_age, and GFS rules with union semantics, and tests cover empty policy, age, bucket selection, and pruned chain membership.
Retention config loading
README.md, docs/RETENTION.md, internal/config/config.go, internal/config/retention_test.go
retention: config blocks, validation, and profile precedence are added, along with README configuration examples.
Prune planning and apply
CHANGELOG.md, README.md, docs/RETENTION.md, internal/dumps/prune.go, internal/app/prune.go, internal/app/prune_test.go, internal/cli/dumps.go
Dry-run pruning now uses chain plans, apply mode deletes pruned chains leaf-inward, and tests cover dry-run, profile scope, collected delete failures, and CLI rendering.
CLI wiring and release notes
CHANGELOG.md, README.md, docs/RETENTION.md, internal/cli/dumps.go
dumps prune now loads config-backed retention policies, applies explicit flag overrides, renders prune results, and the top-level docs and changelog describe the new behavior.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • nixrajput/siphon#1: Directly related pruning/retention work that this PR extends with chain-aware planning and CLI wiring.
  • nixrajput/siphon#7: Touches the same pruning path and Catalog.PruneDryRun surface that this PR rewrites.

Poem

I hopped through chains at dawn’s first light,
Kept fresh incrementals tucked in tight.
Dry runs hop softly, --apply bites,
No orphaned crumbs in moonlit sites. 🐇
Hooray for pruning, neat and bright!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 35.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: chain-aware retention and pruning for retention management.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/phase-g-retention

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@nixrajput

Copy link
Copy Markdown
Owner Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🧹 Nitpick comments (2)
internal/config/retention_test.go (1)

27-53: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add coverage for an explicit empty profile override.

The key contract here is “profile block replaces defaults wholesale.” This test covers omitted vs non-empty overrides, but not the retention: {} case that disables inherited defaults for a single profile.

🧪 Suggested test shape
 func TestEffectiveRetention_Precedence(t *testing.T) {
 	def := &RetentionConfig{KeepLast: 7}
 	prof := &RetentionConfig{KeepLast: 30}
+	emptyOverride := &RetentionConfig{}
 	cfg := &Config{
 		Defaults: Defaults{Retention: def},
 		Profiles: map[string]ProfileConfig{
 			"prod":    {Name: "prod", Retention: prof},
 			"staging": {Name: "staging"}, // no override
+			"dev":     {Name: "dev", Retention: emptyOverride},
 		},
 	}
@@
 	if got := cfg.EffectiveRetention("unknown"); got == nil || got.KeepLast != 7 {
 		t.Errorf("unknown-profile effective = %+v, want defaults", got)
 	}
+	if got := cfg.EffectiveRetention("dev"); got != emptyOverride {
+		t.Errorf("dev effective = %+v, want explicit empty override", got)
+	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/config/retention_test.go` around lines 27 - 53, Add a test case in
TestEffectiveRetention_Precedence to cover an explicit empty retention override
on a profile, using Config.EffectiveRetention and the existing prod/staging
setup as reference. Create a profile with Retention set to an empty
RetentionConfig and assert it returns a non-nil empty result instead of
inheriting Defaults.Retention, while omitted retention still falls back to
defaults. Keep the assertions aligned with the current contract that a profile
block replaces defaults wholesale.
internal/app/prune_test.go (1)

178-194: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a multi-member failure regression here.

This only covers a single-dump chain, so it will not catch the case where a failed leaf delete is followed by parent deletion. Seed base -> inc1 -> inc2, fail inc2.meta.json, and assert the older members are left intact.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/app/prune_test.go` around lines 178 - 194, Add a regression in
TestPrune_CollectsDeletionFailures for a multi-member chain: instead of only the
single dump case, seed a base -> inc1 -> inc2 sequence, configure the store to
fail deleting inc2.meta.json, and assert Prune collects the failure without
deleting the older parent members. Use the existing helpers like
newRecordingStore, seedDump, Prune, and pruneDeps to locate and extend the test.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/app/prune.go`:
- Around line 84-90: The deletion loop in applyChainDeletion continues after a
failed d.Dumps.Delete call, which can leave an incremental orphaned after a
.meta.json failure. Change the loop so that once the first member deletion
fails, it records the error on ChainOutcome/PruneResult and stops deleting any
older members in that chain, preserving the complete-shorter-chain behavior.

In `@internal/cli/dumps.go`:
- Around line 162-175: The dumps renderer is marking every applied outcome as
fully “pruned” even when `oc.Errors` indicates a partial failure. Update the
logic in `RenderDumpPrune` so the verb only becomes `pruned` for outcomes that
succeeded without errors, and use a different message for partially failed
chains while still listing `oc.Errors`; keep `Apply`, `res.Outcomes`, and
`oc.Pruned`/`oc.Errors` as the key checks.
- Around line 85-109: Validate the CLI retention override values before invoking
app.Prune in the dumps command flow, since the policy built in the CLI can
currently accept invalid negative values from flags. Add upfront checks around
the override handling near resolveRetentionPolicy and the
keepLastSet/maxAgeSet/gfs*Set assignments so destructive pruning fails fast with
a clear error instead of passing an undefined policy into app.Prune. Keep the
validation aligned with the existing config-backed retention rules and ensure
invalid overrides are rejected before buildDeps and app.Prune are called.

In `@internal/config/config.go`:
- Around line 61-64: Reject negative retention.max_age values in the config
validation path: in the parsing logic for r.MaxAge, after time.ParseDuration
succeeds, add a check that the resulting duration is not less than zero and
return a validation error if it is. Update the validation in the config reader
around r.MaxAge so it still accepts valid durations but explicitly rejects
signed negatives like -1h, keeping the existing error handling style and the
same retention.max_age validation flow.

In `@internal/dumps/retention.go`:
- Around line 81-90: The chain ordering in GroupChains is only sorted by
newest() after building from a map, so equal timestamps still follow randomized
map iteration order. Update the newest-first sort on Chain (and any matching
sort in Plan) to use a stable secondary key like Chain.Root when newest() ties,
or remove the duplicate Plan sort if GroupChains already guarantees determinism.
Use the existing Chain.newest() comparator and the Root field to make keep_last
and GFS selection deterministic across runs.
- Around line 65-70: GroupChains is currently ordering Chain.Members by Created
alone, which can break the base-first/apply-order contract. Update the
grouping/sorting logic in GroupChains to build each chain’s order from
ParentID/BaseID topology first so the root/base always comes before its
incrementals, and use Created only as a secondary tie-breaker for ties. Keep the
Chain.Members ordering consistent with the documented behavior and the
prune/apply path expectations.

In `@README.md`:
- Around line 184-194: The prune command name in this README example is
incorrect and should match the documented CLI path. Update the retention section
text to use the actual command name from the commands table, `siphon dumps
prune`, and make sure any related mention of dry-run or `--apply` refers to that
same command so users are pointed to the right entrypoint.

---

Nitpick comments:
In `@internal/app/prune_test.go`:
- Around line 178-194: Add a regression in TestPrune_CollectsDeletionFailures
for a multi-member chain: instead of only the single dump case, seed a base ->
inc1 -> inc2 sequence, configure the store to fail deleting inc2.meta.json, and
assert Prune collects the failure without deleting the older parent members. Use
the existing helpers like newRecordingStore, seedDump, Prune, and pruneDeps to
locate and extend the test.

In `@internal/config/retention_test.go`:
- Around line 27-53: Add a test case in TestEffectiveRetention_Precedence to
cover an explicit empty retention override on a profile, using
Config.EffectiveRetention and the existing prod/staging setup as reference.
Create a profile with Retention set to an empty RetentionConfig and assert it
returns a non-nil empty result instead of inheriting Defaults.Retention, while
omitted retention still falls back to defaults. Keep the assertions aligned with
the current contract that a profile block replaces defaults wholesale.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 84741d26-28e7-45f4-830f-9d61aa016d6e

📥 Commits

Reviewing files that changed from the base of the PR and between 01d23f2 and ce22097.

📒 Files selected for processing (11)
  • CHANGELOG.md
  • README.md
  • docs/RETENTION.md
  • internal/app/prune.go
  • internal/app/prune_test.go
  • internal/cli/dumps.go
  • internal/config/config.go
  • internal/config/retention_test.go
  • internal/dumps/prune.go
  • internal/dumps/retention.go
  • internal/dumps/retention_test.go

Comment thread internal/app/prune.go Outdated
Comment thread internal/cli/dumps.go
Comment thread internal/cli/dumps.go
Comment thread internal/config/config.go
Comment thread internal/dumps/retention.go Outdated
Comment thread internal/dumps/retention.go
Comment thread README.md Outdated
- Apply post-review fixes to the retention cycle.
- Order chain members topologically (base first, each child after its
  parent via ParentID), with Created only as a tie-breaker. Ordering by
  Created alone could place a child ahead of its base on tied or skewed
  timestamps, and the leaf-inward delete then removes the base before a
  surviving incremental — orphaning it. Stragglers from broken/cyclic
  links are appended, never dropped.
- Stop deleting a chain's members at the first failure (return, not
  continue): once a leaf delete fails, deleting its ancestors would
  orphan it, so abort the chain and leave a complete shorter prefix.
  Other chains still proceed.
- Make chain ordering deterministic: break equal newest() timestamps
  by Root so keep_last / GFS selection is reproducible (map iteration
  is randomized; a planner must not be). Same comparator used in Plan.
- Reject negative retention.max_age in config validation: ParseDuration
  accepts "-1h", which would push the cutoff into the future and make
  the whole catalog deletable.
- Reject negative CLI flag overrides before planning, mirroring the
  config-load validation that flags otherwise bypass.
- Label a partially deleted chain "partially pruned", not "pruned".
- Fix the command name in README and docs/RETENTION.md: it is
  "siphon dumps prune", not "siphon prune".
- Add tests: topological member order on tied/skewed timestamps,
  deterministic chain order on tied timestamps, and that a chain's
  deletion stops (base untouched) when the leaf delete fails.
@nixrajput nixrajput merged commit abed659 into main Jun 25, 2026
5 checks passed
@nixrajput nixrajput deleted the feat/phase-g-retention branch June 25, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant